Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make a shallow clone of git submodules #24521

Closed
wants to merge 1 commit into from
Closed

Make a shallow clone of git submodules #24521

wants to merge 1 commit into from

Conversation

yberreby
Copy link

When building from source, the build system makes a deep clone of LLVM and other dependencies, downloading several hundred megabytes, which takes a significant time over a slow connection. A shallow clone (done with git submodule update --depth 1, see http://stackoverflow.com/a/17692710/2754323 ) would only download the latest revision of the dependencies, since the rest of the commit history is not needed, and save bandwidth and space at no (as far as I know) cost.

@rust-highfive
Copy link
Collaborator

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. The way Github handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see CONTRIBUTING.md for more information.

@yberreby
Copy link
Author

r? @steveklabnik

@rprichard
Copy link
Contributor

I tried running git submodule init and git submodule update --depth 1 in a new Rust checkout, and it didn't work. The update command downloaded the master branch of https://github.com/rust-lang/llvm.git (commit 2e9f0d21fe321849a4759a01fc28eae82ef196d6), which is two years out of date. Then update complained that it couldn't find the commits it expected:

rprichard@ryan:~/work/rust-test$ git submodule update --depth 1
Cloning into 'src/compiler-rt'...
remote: Counting objects: 1405, done.
remote: Compressing objects: 100% (1226/1226), done.
remote: Total 1405 (delta 289), reused 845 (delta 152), pack-reused 0
Receiving objects: 100% (1405/1405), 2.02 MiB | 3.67 MiB/s, done.
Resolving deltas: 100% (289/289), done.
Checking connectivity... done.
fatal: reference is not a tree: 58ab642c30d9f97735d5745b5d01781ee199c6ae
Cloning into 'src/jemalloc'...
remote: Counting objects: 164, done.
remote: Compressing objects: 100% (145/145), done.
remote: Total 164 (delta 17), reused 80 (delta 14), pack-reused 0
Receiving objects: 100% (164/164), 354.07 KiB | 0 bytes/s, done.
Resolving deltas: 100% (17/17), done.
Checking connectivity... done.
fatal: reference is not a tree: e24a1a025a1f214e40eedafe3b9c7b1d69937922
Cloning into 'src/llvm'...
remote: Counting objects: 10756, done.
remote: Compressing objects: 100% (10239/10239), done.
remote: Total 10756 (delta 1194), reused 4076 (delta 433), pack-reused 0
Receiving objects: 100% (10756/10756), 14.39 MiB | 6.03 MiB/s, done.
Resolving deltas: 100% (1194/1194), done.
Checking connectivity... done.
fatal: reference is not a tree: bff69076975642c64e76dbeaa53476bfa7212086
Cloning into 'src/rt/hoedown'...
remote: Counting objects: 83, done.
remote: Compressing objects: 100% (81/81), done.
remote: Total 83 (delta 1), reused 62 (delta 1), pack-reused 0
Unpacking objects: 100% (83/83), done.
Checking connectivity... done.
fatal: reference is not a tree: 238c4d57cce10d33b05cf52a91fc62a09f31ffbb
Cloning into 'src/rust-installer'...
remote: Counting objects: 27, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 27 (delta 2), reused 18 (delta 0), pack-reused 0
Unpacking objects: 100% (27/27), done.
Checking connectivity... done.
fatal: reference is not a tree: ebc6b04c29591108d3f28e724b4b9b74cd1232e6
Unable to checkout '58ab642c30d9f97735d5745b5d01781ee199c6ae' in submodule path 'src/compiler-rt'
Unable to checkout 'e24a1a025a1f214e40eedafe3b9c7b1d69937922' in submodule path 'src/jemalloc'
Unable to checkout 'bff69076975642c64e76dbeaa53476bfa7212086' in submodule path 'src/llvm'
Unable to checkout '238c4d57cce10d33b05cf52a91fc62a09f31ffbb' in submodule path 'src/rt/hoedown'
Unable to checkout 'ebc6b04c29591108d3f28e724b4b9b74cd1232e6' in submodule path 'src/rust-installer'

The current branch is rust-llvm-2015-02-19, but .gitmodules specifies the master branch instead.

I'm using git version 1.9.1 on Xubuntu 14.04. Maybe that matters?

@pnkfelix
Copy link
Member

@rprichard I think what you describe is consistent with @brson's findings when he investigated trying to do this, based on a recent conversation I heard him having.

@yberreby
Copy link
Author

After research, this seems to be a common issue with submodules and low values of depth:
CocoaPods/cocoapods-downloader#32
https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/EJtj5W-60-Y
http://www.tech.theplayhub.com/git_submodule_head_reference_is_not_a_tree_error/

@alexcrichton
Copy link
Member

I believe a bare --depth 1 means that git will fetch the master branch with a depth of 1, but the LLVM repo that we're using has a branch for each rev that we're pinned to. This means that the clone will have to specify both the branch and the depth, I believe.

@steveklabnik steveklabnik assigned brson and unassigned steveklabnik Apr 21, 2015
@brson
Copy link
Contributor

brson commented Apr 22, 2015

As others have said, Rust submodules are not using the master branch of their repos, so a shallow clone of master will not work.

The best way I can think to get to a point where this works would be to set all our submodules to a stable named branch and state that branch in .gitmodules, make it a policy matter that submodules always use that name.

Getting there will require further planning, so closing this PR.

@brson brson closed this Apr 22, 2015
@brson
Copy link
Contributor

brson commented Apr 22, 2015

If you want to pursue this further I might suggest opening up a thread on internals.rust-lang.org laying out a plan.

@brson
Copy link
Contributor

brson commented Apr 22, 2015

Oh, the plan I stated previously doesn't work because historic checkouts don't work after submodule master branches have changed.

@orbea orbea mentioned this pull request Sep 30, 2017
Closed
@pramodk pramodk mentioned this pull request Mar 25, 2019
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@brson brson

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@yberreby @rust-highfive @rprichard @pnkfelix @alexcrichton @brson @steveklabnik